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(57) ABSTRACT 

An infrared detecior having two cameras for viewing a scene 
to deled humans. The cameras focus Lhe scene on their 
respective pixel arrays. One array 15 designed Willi a filter io 
be sensitive io %hi from 08 in 1 .4 microns and the array of 
the other camera is designed with a filter to be sensitive from 
1.4 microns to 2.2 microns- The arrays are co-Tcgislered 
Specially and temporally relative lo the viewed scene and 
pixel transmission dynamics. The spectral powers of the 
pixels from both arrays are difference weighted and fused 
into one sei of pixels. The fused pixels arc ibresholded into 
• another set of pixels that have one of Iwo values — black or 
while. These pixels are entered into a visual display ihai 
shows a segmeni image of a hum an if in the viewed scene. 

37 Claims, 7 Drawing Sheets 
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NEAR-IR HUMAN DETECTOR 
BACKGROUND 

The invention pertains 10 dclcclioa Of people, and par- 
ticularly lo detection of occupants in vehicles. More 
particularly, II pertains 10 image fusion in the near-infrared 
band under various kinds of environmental conditions. 

The gathering of usage statistics in tbo high occupancy 
vehicle (HOV) Ibdc is desired by wme go^crairwni agen- 
cies. These statistics arc crucial for construe lion planning. 
Currently, the gathering of data is performed manually. This 
approach is obviously laborious* inefficient, and prone to 
error. 

There arc compelling reasons for the existence of an 
automatic occupant counting system Jn the IIOV lane. It 
would facilitate Ibc gathering of statistical date for road 
construction planning. It would enable State authorities to 
charge a nominal fcc to single occupant vehicles in HOV 
lanes. U would also help the state patrols lo perform their 
monitoring tasks more crfcctivcty- 

The occupant counting system needs to be reliable. In a 
sensing system, reliability is randy achieved if the signal is 
corrupted with noise. The first concern in the present effort 
& to produce a signal with as distinct » signature for the 
vehicle occupant as possible. This goal can be achieved only 
through careful design and arrangement of the sensing 
elements. 

If one manages to acquire a clear imaging signal through 
sensors, then even moderately powerful pattern recognition 
algorithms could accomplish the occupant detection task. If, 
however the imaging signal were noisy, then even the most 
pc^verfiil pan era recognition algorithms could not accom- 
plish the Usk. 

Related efforts by others have involved the use of a 
ueflT-iufrared Camera (0.55 lo 0.90 micron) and a near- 
infrared illumination source in the same range of wave- 
lengths. One reason for using Dcar-inJxared sensing was Ihe 
ability lo use non-distracting illumination at nigbL lHuini- 
nalion at nighttime enhances the quality of the image. 
Hoveve^ it appears that the choice of range of wavelengths 
is not appropriate because of its close proximity to the 
visible spectrum. Psychophysical experiments have shown 
that the bum-tin eye has some sensitivity to this range of 
near-infrared wavelengths, however small This sensitivity 
may be suflicieni to potentially cause accidents voder certain 
conditions. Another reason for this approach, according to 
others, was to bypass the problems caused by solar iUumi- 
nation during daytime, such as glare. Nevertheless, particu- 
larly in Ih at range (i.e., 0.55 lo 0.9 micron) solar ilfumination 
is frill substantial and the associated glare can be reduced so 
only through the use of polarizing Oilers. 

In more general terms, related art projects that involve 
imaging usually adopt the use of visible spectrum cameras. 
The strong point of the visible spectrum approach is that the 
relevant imaging sensors are very advanced and at the same 55 
lime the cheapest across the electromagnetic (EM) spec- 
trum. Visible spectrum cameras have a particular advantage 
in terms of speed, which is an important consideration in the 
HOV lane where vehicles are moving at rates of speed of 65 
mpb. These cameras can also have very high resolution, 60 
resulting in very clear images under certain conditions. 
Unfortunately,, there are serious problems with the visible 
spectrum approach. For instance, some vehicles have 
heavily tinted window glass to reduce glare from solar 
nomination. This glass is Dearly opaque to visible spectrum 65 
cameras. Also* visible spectrum cameras do not have opera- 
tional capability during nighttime. 
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Many researchers adopt the visible spectrum as the spec- 
trum of choice, or, in rare cases, some other EM spectrum 
based primarily on intuition. The result is that Ihey usually 
end up with a non^jscrirrrinatidg signal thai makes the 
5 de Lection problem appear more difficult than it actually is. 
Then, they try lo address the difficulty by devising powerful 
pattern recognition algorithms but often to no avail. The toss 
of information because of a poor sensor choice, spectrum, 
and arrangement is usually irrevocable, 
10 Visible spectrum or very near infrared detection of people 
in vehicles has not been very successful under most condi- 
tions. The glare and other problems caused by solar 
illumination, such as through vehicle windows, has pre- 
vented effective detection of vehicle occupants. Also, cuvi- 
35 ron mental conditions like weather obscure detection. People 
appear 10 have darker or lighter faces, depending on the 
characteristics Of the people being delected, and on the 
incident angle and intensity of deliberate or incidental 
iUumination- Other wavelengths of the EM spectrum do not 
20 appear to offer inexpensive, compact, and high resolution 
sensing and detection of human beings In vehicles. 

The lower portion of the EM spectrum consists of the 
gumma rays, the x-rays, and radiation in the ultra-violet 
range. Kadiation of such wavelengths is harmful This 
25 radiation is typically used in a controlled manner in medical 
applications. 

Al Ibe Tar end of the £M spectruzu, there is the microwave 
and radio radiation. This range was recently started lo be 
exploited for imaging purposes- Sensors operate in an active 
or in passive mode- The major advantage of these longer 
wavelengths is ihai they can penetrate clouds, fog, and rain 
for producing weaker-independent imaging results. The 
technology for these wavelengths is new, and prohibitively 
expensive. Also the sensors are bulky in this range of 
radiation, and feature very low resolution. A useful appli- 
cation of these sensors is currently confined to the military 
and the remole-sen&ing domain. 

SUMMARY OF T1IE INVENTION 

The present invention utilizes radiation in I be middle 
region or the EM spectrum regarded as the infrared spec- 
rrurn. This spectrum includes wavelengths from 0.7 lo 100 
microns. Within the infrared range, two bands of particular 
interest are ibe 0.7 to 3.0 micron, 3.0 to 5.0 micron and 8.0 
lo 14 micron bands. The latter two bands are regarded as the 
thermal infrared band and the first band as the reflected 
infrared band. The reflected infrared band is associated with 
reflected solar radiation that contains no mforxnalion about 
the thermal properties of materials. This radiation is for the 
most part invisible to the human eye. The thermal infrared 
band, on the other hand, is associated with the thermal 
properties of materials- 

The thermal infrared band is significant for several rea- 
sons. First, the human body maintains a relatively constant 
temperature of about 37 degrees Celsius (C), irrespective of 
physical characteristics or illumination conditions. This 
indicates a consistent light color pattern for the faces of 
vehicle occupant* subject to thermal infrared imaging. This 
consistency is lacking in the visible specu*um. Such consis- 
tency facilitaies interpreting sensed images. Further, (ho 
therm al property serves as a differentiator between humans 
and dummies. Also, a sensor functioning in the thermal 
region is operational day and niuht without any need lor an 
external ulummation source. 

However, one concern is the attenuation of therm*! Infra- 
red radiation caused by glass, when detecting humans in a 



PAGE 20134 ' RCVDAT 111912004 4:00:32 PM [Eastern Standard Time] ' SVR:USPT0-EFXRF-1(8* DNIS:8729306 4 CS1D:6123051228 ' DURATION (mm-ss):11-18 



11/08 /2004 15:03 FAX 6123051228 



MUETING RAASCH GEBHARDT 



©021 



US 6,37 

3 

vehicle. The glass severely disrupts Ibe transmission of 
infrared radiation at wavelengths greater than 2.8 micTuiw. 
At 2.8 microns, thermal energy jt»5l begins TO appear. To 
obtain an infrared image under such conditions, one needs a 
very sensitive mid-infrared camera in tbe range froni 2-0 to 
3.0 microns. Vehicle windows are not made from common 
glass £6r reasons of safety* energy efficiency, and visibility. 
AJso, the composition of ibe front windshield differs sig- 
nificantly (totd the composition of the Side windows of a 
vehicle. The side windows are more transparent to the 
transnisnon of thermal infrared radiation. However, detec- 
tion wiih a near-infrared camera significantly reduces this 
problem of radiation a Herniation. 

A near-infrared camera, if it is restricted to the appropriate 
range, outputs similar imaging signals for Various humans 
despite their having different colors of skin. However, Ibis 
camera outputs a much different imaging signal for a dummy 
having the same visible color as the human skin. 

One embodiment, of i be present invention has two cam- 
eras of diJeTeni sensing wavelengths in ibe near-infrared 
bandwidth. These cameras are pointed toward a place where 
humans may be detected. A near-infrared lamp for the 
jj rumination of tbe scene may be used. Tbe rwo outputs of 
the cameras arc fused together with a weighted difference to 
result in an image having, an intensified contrast. The image 
output of the device thai performs ihc fusion of 1be two 
camera images goes to a rx*si-processor t which performs 
binary trnesbolrling oo the various pixels of tbe fused image. 
The result is an Image with each pixel cither being black or 
White. Ibe thresholdcd ooipui undergoes such operations as 
fuzzy neural network or analytical processing. The thresh- 
oldcd output diminishes all Of the background of the viewed 
scene, except human skin such as races. This approach is one 
embodiment of the burn an detector. 

BRIEF DESCRIPTION OF THE DRAWING 

FJG. 1 shows a near-infrared fusion syslcm lor detecting 
humans. 

FIG. 2 illustrates a co-registration scheme for rwo cam- 
crass. 

FIG. 3 is a histogram of the number of pixels versus 
spectral power for an image frame. 

FIG. 4 is the graph of the EM spectrum. 

FIG. 5 reveals the infrared uansTpiitance characteristics 
for on automobile windshield. 

FIG. 6 reveals the infrared transrnittance characteristics 
for an automobile side window. 

FIG. 7 is a graph showing the percentage of reflectance 
for Caucasian males of light and dark complexions- 

FIG. 8 is a graph showing the percentage of reflectance 
for Asian males of light and dark complexions. 

FIG, 9 is a graph showing the percentage of reflectance 
for black males Of light and dark complexions. 

FIG, 10 is a graph showing a comparison of reflectances 
for light and dark skin. 

FIG. 11 ia a graph of reflectances Tor cotton, wood and 
poly amide. 

FIG. 12 is a graph of reflectance for distilled water. 
FIG. 13 shows a layout for dcicnrdjrmg the speed char- 
acteristics of a human detection system. 

DESCRIPTION OF THE EMBODIMENT 

FIG. 1 shows a basic layout of a near-infrared fusion 
system 10 for detecting humans. It is a dual-band imaging 
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system. Two co-registered cameras 11 and 12 sense the 
image of, for instance, a face 13 of a human being. Camera 
11 has a spectral sensitivity of 0.& to 1 .4 microns- Camera 12 
has a spectral sensitivity of 1-4- to 2.2 microns. Slightly 

5 shorter or longer ranges can also yield acceptable detection 
re&ulis. The 1.4-micron Threshold point between the rwo 
bands of spectral sensitivity is a preferable demarcation 
point for the dual-band system 10, but maybe another value 
as appropriate. Each of the camera sensitivity band ranges 

1D can cross somewhat tbe 1.4 micron wavelength without 
diminishing tbe human detecting ability of system 10. 'Ibe 
quality of the imagrng signals from cameras U and 12 
remain high even during overcast days and ai nighttime, 
because the scene being scanned by cameras 11 and 12, can 

j j. be iUiiiniOal cd wi | b an eye -safe ncar-jnlrared illu mtn a Lor 14. 
Since the eye js not {Sensitive to the near-infrared spectrum, 
system 10 can remain stealthy all of the lime, whether in a 
detection mode or not. Ideally, the camera at the lower band 
range (0.8 to 1.4 microns) should be an image intensjfier. 

10 Therefore, the illuminator's spectral emission specification 
needs to match only with the upper band range (1-4 to 2.2 
microns). The upper band range is quite far from the visible 
spectrum and illumination in these wavelengths is safe even 
for highway applications. Near-infrared cameras 11 and 12 

25 provide Clear irr aging, signals even in foul weather condi- 
tions such as hazy conditions. These particular infrared 
bands of detection and illumination provide fur sufficient 
light transmission through windshields, side windows, fog, 
and darimesK. This permits adequate detection of humaoS in 

30 vehicles at night and in poor weather 

The image outputs 15 and 16 of cameras U and 12, 
respectively, go to a weighted difference software process 17 
or specialized hardware that fuses the outputs by performing 
weighted subtraction nf the intensities of the two camera 

35 images. This weighted difference software or hardware may 
be referred to as a fuser. Such fusion of the camera outputs 
intensifies the silhouette of face 13 and other exposed human 
skin In tbe resultant fused image. Also, the image tea Lures a 
diminuiion of the background of the scene being covered by 

40 the cameras* This Increased contrast between the person and 
the background in tbe fused image permits essentially per- 
fect image segmentation through thresholding by a software 
process 18, or specialized hardware. This thresholding soft- 
ware or hardware may be referred to as a thresholder- The 

45 output of the thresholder J ft may go to a display 19, printer, 
or a post-process or specialized hardware. 

A final processed image shows the exposed skin parts, 
such as face 13, as binary blob 13 r , as shown in FIG- 1- 
Backgmund 20 of sensed face 13 is discounted as shown by 

50 blank background 20* in display 19. This clear-cut binary . 
imagery ensures reliable and fast operation of a partem 
recognition algorithm that identifies a human as indicated by 
face 13' imagery- 
FIG . 2 illvstra tes the co-registratioTi of cameras 11 and 12. 

55 There isspacial and time registration between the cameras. 
The cameras could be of the same make and model The 
necessary difference between the cameras is the optical 
bandwidth filters, 48 and 49. which are situated between 
sensing arrays 46 and 47 and Camera lens 58 and 59, 

60 respectively, of cameras 11 and 12- Filter 48 determines the 
0-8 to 1.4 micron spectral sensitivity of array 46 in camera 
11 and filter 49 determines the 1.4 to 2.2 micron spectral 
sensitivity of array 47 in camera 12. A polarizer may be 
inserted in front of fens 58 of camera 11 and in front of 1cm 
es 59 of camera J 2. Or instead, a polarizer may be inserted 
between lens 58 and array 46 of camera 11, and between lens 
S9 and array 47 of camera 12. Sensing arrays 46 and 47 of 
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ihe cameras arc lb* same size, for example, 512 by 512 fused pixels- Processor specialized hardware 18 emulates a 

pixels in a EolliuiD arsenide substrate. TVpically, the fields of comparator-like circuit in that each fused pixel below a 

view art the same for each array- Three pixels 51 , 52, and 53, certain threshold value (I) is assigned a value (V,) of 0 and 

for example, are selected mil for spacjal co-registration, each fused pixel above iho certain ihreshold value js 

Each of the three pixels are focused on corresponding s assigned a value (VJ of 255. . _ j 

ponions si\ 52\ and 53\ respectively, of image 20 viewed FIG- 3 shows a histogram of an image tome of fused 

by cameras 11 and 12- That means Ibe arrays have the same pixels The number of pixels ibf each spectral power value 

images pixel for pixel, even though the Spectral sensitivities >* indicated by curves 54 and 55 for a given sensed image. 

lii* diffcrenL In other words, the columns and rows of pixels. The pixels of curve 54 represent background 20 and trie 
are aligned with the physical world scene, pixel for pixel. 10 pixels of curve 55 represent human sfcin 13. Curves 54 and 

Once spacially co^egistercd, cameras 11 and 12 are kepi S5 intersect at 56 thai is deemed m be the appropriate value 

stationary relative lo the physical world. for thresholding If curves 54 and 55 do not intersect, then 

Tune co-TCgisiraiion of the cameras means that the cam- ^holding value 56 u. centered between curves 54 and 

eraT^^chronization with each Other from a signal » H» ^hold value * dynamic ^AW>£ 

perspective The signals for each of the two corresponding is frame to frame, and is determined for each image frame 

£X£7o aT^Suffcr at the same time. The mention » ^logramof the re^ec ivc frame. Utto 

Klforeachp^ ^ J* it^oidmg m 20 pixels 

Same time is about 33 milliseconds, which is 30 frames per having, a value below 20 are vahied at 0 and itaed] p»b 

second. The transfer of pixel data may be parallel, linc^y- having a value above 20 are valued at 255- The resulting 

U^7ririaLp^ 20 i*agcind,splayl9 D sswm^^^ 

innate and mainiain their lime cc-registralion. procc^ or spemU^d hardware lSsucb thai background 20" 

" . f . t~,*r ~, * has black pixels and face 13* has wtnte pixels. 

Tne image outputs or pixel signals ^ ^ ™ As one can note here, a main application of the invention 

software process or specialized haidwarc 17 which proyicto ^ & dclliClioa m vehicle*. However, people detection 

a certain weighting to each Ot the pixels and L fuses pair* of J^*^ M ^ ^ ^ SO forth. An 

corresponding mxdsfrom c«r*™ U and 12, re^e a rvely^ ta ^ aMd of sysicm 10 may be used <o actually 

into single pixels. The weighted differencing n> performed klcat5£v Jb 

pixel by pixel. Each result is the fusee I pixel of the two ^c^^ms that cameras 11 and 12 function are within 

weighted differenced pixels. The weight difference equation ^ (q& reflected infrared portion 21 In FIG. 4, which shows the 

for such fusing is £M .spectrum. Visible fipectram 22 is the spectral sensitivity 

bi, a ~»r. n j?»P nn _ of conventional cameras. Unformnatcly, visible light cam- 

PluWFCiJ)/— b a V c nofec levels that increase during poor cnvuon- 

P is spectral power. The position or location of each pixel in mental conditions such as bad weather, nighttime, and direct 

the respective image is identified by row (i.e., "i") and 35 sunHgbt. Some problems, such as nig^rtfme viewing may be 

column (i.e., "}'% The rows and columns of pixels of the overcome with artificial lighting, which matches The visible 

images of cameras 11 and 12 coincide with each other. The spectrum of the camera, and which in vehicle occupant 

lower band pixels are these from camera 11 and the upper detection is a serious distraction to drivers. Another disad- 

band pixels are Those from camera 12, The spectral power vantage is thai a human face 13, which is the object of 

"V for each pixel at ij is indicated by a numeral scale of 40 interest, does not have consistent qualities within the visible 

brightness from 0 to 255. ii 0" is entirely black or dark (i-e., range. Vehicle occupant faces appear dark or ligbi, dc 

no spectral power) and "255"' is entirely while or bright (ix., pending o n the physiological characteristics of the 

foil spectral power). Numerical indications in between 0 and occupant, and the intensity and incident angle of iUumina- 

255 are, of course, representative of various gradations of uon. 

gray, brightness, or Spectral power. "C of the equation is a 45 The thermal infrared bond 23 (3.0 to 5-0 and BX> to 14 

constant (i.e., weigh ling factor), which is determined microns) is associated with thermal properties of materials, 

according to the Ughting of background or scene 20 and Tne human body is at a tcmpcralure of 37 degrees C This 

object or face 13. **C* for daytime lighting conditions is means thai human faces have a consistent lighl color in 

about 3 for optimal resnlLs. **C* for mgtmime depends upon thermal infrared imaging, despite various facial colors, 

the spectral distribution and properties of an artificial illu- 50 which is contrary to visible imaging, 

minator 14. The Thermal property of the body provides a clear difltr- 
Tbe brightness or spectral power of a typical pixel of the 1 cntialor from look- alike dummies. The thermal infrared 

lower band (image 4*) may be 55 and the brightness or sensor can function at night without an external illuminator, 

spectral power of the corresponding typical pixel of the One disadvantage of using (be thermal infrared band 23 for 

upper band (image 47) may be 10. Tbese upper and lower 5$ occupant detection is thai vehicle windshield glass greatly 

band pixel values are representative of dan in the cone- attenuates infrared light transmission at 2.8 microns and 
spending bands. The spectral power of a resulting typical higher. 

fused pixel, in daytime viewing, ai a location of row i and FIG, 5 reveals the tr admittance characteristics of infrared 
column j in ihe daytime is determined wilh the following light between 0.4 and 2-8 microns for a clean vehicle 

calculation. 60 windshield (curve 24) and a dirty windshield (curve 25). 

Beyond 2.8 microns of thermal infrared bandwidth 23, the 

pCmUw^ 55 - 3 " 10 radiation traiismiiiBnce characteristic of windshield glass is 

almost zero. The transmit tan cc of a lightly timed side 

rtmk^s- window of a vehicle is good (50 to 85%). from 0.3 up to 2-8 
The fused pixel signals uO from software process or tf5 microns as reveal cd by curve 26 In FIG. 6. Between 2.8 and 
specialized hardware 17 lo software process or spedaHzed 4.3 microns, the radiation uansmillance is about 20 perceni 
hardware 18 for image seymenlution via thresholding of the for the side vehicle window. Beyond 43 microns the trans- 
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mitiancc drops 10 nearly zero. However, The Spectra] behav- Abo, since cameras 11 and 12 or system .10 operate at a 

ior of the side window permits transmit ai>ce of some lower band than the mid-infrared band, glass penetration is 

inenrjaj radiation, not a problem and cameras 11 and 12 can easily delect 

Curves 27 and 18 of FIG, 7 show the pcrccntafiC of through the frontal windshield of a vehicle. IbuS, Speed 

reflectance of infrared light for light and dart complexion s requirements for cameras 11 and 12 arc less restrictive, la an 

Caucasian males, respectively. The reflectance i» good { highway silo, a zoom lens would be uscd- 

between 0.6 and 1.4 microns. Above 1.4 microns, the * J 9 

reflecUoU is significantly diminished. However, ibe differ- FIG. 13 shows a layout of a near-infrared system 40 for 

ence of reflectances of light and dark complexions; are dclcrmining the speed characteristics of the human detector, 

minimal. 10 A vehicle 41 may be assumed to be moving down a highway 

In FIG. 8, curves 29 and 30 show skin reflectances for at a velocity v, as shown by vector 42, and be observed io 

light aod dark complexions, respectively, of Asian males. a frontal view with a near-infrared camera 11 or 12 at a 

Curves 31 aod 32 of FIG. 9 show skin reflectances for light distance d, as shown by line 43. and from a height h, as 

and dark complexions of black males. Even though the shown by ^ 44 Qnlv onc of ^ cameras 11 and 12 is 

refleci^ces of light comple^ons a/e higher than those of ^ ^ bui ^ m ^ eimer 0QC of ^ 

a^kcc^l^curve,21 •»*^ 1 JJ^^ F1 ^ Camera 11, 12 may be a Sensors Unlimited lac. SU 320 

7-9. respectively, have sjorilar shapes and all 01 Incm drop ' ' 

off J Xl 1.4 microns. These reflectances Show some -qmpP^ with a lelephoto lens, a band-pass filter in the 

variation for all complexions of the Caucasian, Asian, and range 1.4 10 x microns (where x>l.4 microns), and a 

black males, between U-6 and 1.4 microns. polarizing filler to reduce the glare effect from the sun 

Curves 33 and 34 of FIG. 10 show the reflectances for 20 illumination during daytime, 

more extreme differences of light skin and dark skin, reqwc- During the daytime, system 40 uses the fllummaiian of the 

tivcly. The reflectances of light and dark skin arc significant sun objective is tn determine if there is any appropriate 

up to 1.4 microns. Aflec LA microns, reflectance curves 33 geometric arrangemenl for camera 11, 12 so that the signal 

and 34 for light and dark skfn. n^Uvely, rmcor^ almost ^ ^ ^ ^ ^ ^ camcra Speed ^ ^ at 

coincident and the resultant reflectances drop below 20 is acccpUblc ]evels ^ cu undcr ^vc,^ conditions. An accept- 

percent Thus, in the nej-irJrared a We (S/N) ratio is considered anything above 35. The speed 

tbc reflectance of dented humans ^ ^ 0 f^^ qnab'ty is considered acceptable when the image smearing 

arc about ibe same at wavelengths greater than 1.4 microns. , , , . . AtU * r 

In WO. 11, eurvesSS, 36, and 37show the reflectance for does «* exceed the width oC one piXCl. 

cotton, wood, and polyamidc respectively, to be significant 30 The first step in a radiometric computation is lo deiermme 

not just between 0.6 and 1-4 microns, but beyond 1.4 the amount of radiation that falls upon the objects of interest 

microns- The insigniflcant drop in reflectance after the 1.4 such as the occupants of vehicle 41- The spectral band 

micron threshold point, is a basis that shows a large contrast considered is above the 1.4-micron Threshold point. Because 

in reflectance between the human face and inanimate of constraints due to the quantum efficiency of the camera 

Objects, such as upholstery, the dashboard, and fabrics in a is SU-320 that was used in the actual experiments, one limits 

vehicle, which arc background, and provide for easy deleC- ^ spectral band in the range of 1.4 lo 1 .7 microns. Slightly 

lion of the human Ikec in the range of 1.4 to 2-2 microns. modified things arc in effect for the extended range of 1 -4 to 

Consequently, there is a significant contrast, in reflectance ^2 nucronH. The spectral irradiance of the sun (the fflumi- 

botween the images of a Caucasian male and a dummy head natj0n an a clear day ai sea level is appnwimalely 

from a camera operating in the range beiweco 1.4 and 2.2 <u J 0 J008 Watts/cm* in the 1.4 to 1.7 micron band range, 

microns- The image of the dummy is reflective and appears m ^ paring however, onc considers the worst case 

rather bnghl and the male human image is dark and thus the 1 fgT m ^ the 

resultant contrast between the two images * rather stark. ^ fc , ^ m 

This person dctecnoo scheme is much superior to visible ^ approximately 

light imaging, since the latter scheme reveals hide or no 45 TCraac * A OL ^ 

contrast between the Caucasian and dummy heads of like _ 

colors. In summary, il is easy to distinguish images of the " 1(r 

human head from loose of the durmny head in the 1 .4 to 22 _ l0 -3 :i Q_ cna 
micron range imagery, but not easy 10 distinguish the images 

of the respective heads from each other fn the 0.8 tn 1.4 50 =8 uw P ii*fcm*. 
micron range Imagery. 

The lower reflectivity of human skin for the 1.4 to 2.2 . 

micron spectral range is explained by the spectral reflectance ^ transmiiiance In this spectral range of windshield 45 

of distilled water as shown by curve 38 of FIG. 12- There is of vehicle 41 is approximately 0.4 resulting in an irradiance 

a substantial drop in reflectance at about 1.4 microns. 55 on the vehicle occupants of 
Beyond 1.4 microns, the water absorbs substantial infrared 

radiation and appears in an image as a dark body. Since the = 0.4 * Uim 

composition of the human body consists of 70 percent water -oa*k 
naturally, its spectral response is similar 10 thai of water. So 

camera )2, operating in the 1.4 to 22 naicron range, captures 00 = 33. pWfo*/^. 
this unique human body differentiator. With the operating 
range of camera 12, one can safely use during nighttime a 

matching near-infrared illumination source 14 10 improve The second step in a radiometric compulation is 10 

the quality of the sensed image of race 13. This light is determine how much of the incident radiation on ibe objects 

invisible to humans, such as vehicle drivers, but also us 65 of interest is reflected back to the sensor (ie., near-infrared 

rmlcss tn iheir eyes since ihc wavelength of illuminator 14 camera 11, 12). The radiance into » hemisphere assuming a 

is above the safe threshold of 1.4 microns. reradiate of 0.4 would be 
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a i th c specified distance 43, will achieve an acceptable smear 
Of less Lhan one pixel because the required exposure time of 
1 ms is within tbe camera's Speed capabilities, fhc signal 10 
noise ratio (5/N) is 0.7. lb bona. Ibe S/N ratio to a higher 
Value Oil overcast days, one needs 10 employ ail ilhimjnalion 
source 14. n hirmn ariop source 14 will also be bclpful during 
nighttime. If ore oper aicd in the visible spectrum, the use of 
an illuminator in Ibe high occupancy vehicle (IIOV) lane 
would be prohibitive Fortunately, in this case, ibe spectral 



s 0.000 yWallsfcnr. 



Camera U, 12 has square pixels with a side of 375+1Q- 4 
cm or an area 



30 



This represent the reflected portion of IDS occupant 
irradiation. The occupant's body absorbs the rest. The . , ^ . 

reflected radiation has to pass through windshield 45 and the to signature of Oft ruinator 14 for the J .4 to 1.7 micron wave- 
camera 11, 12 lens to reach the ncar-infrarcd sensor array of band can be safely employed in the HOV lane, 
camera 11, 12- One assumes a 0.4 windshield iransmj nance, Post processing includes a neural network that performs 

a 1/2 camera lens (i.e., having a 1432° cone angle) with 0,8 automatic vehicle oecupam detection. 1 he vehicle occupant 
uansniitunce, a polarizer wiih 0.4 transniittancc, and a detection approach is based upon a fuzzy neural network 
band-pass filler wiih 0.6 transmiuance. Then, the irradiancc as algorithm. The perfect binary image provided by ibe fusion 
at the sensor array of camera 11, 12 will be approach described above facilitates high correct detection 

rates. 

W« = 0.4-0.8TO^»J^^«>U : ««»''^ t (M-32 fl ) What is claimed is: 

1. A human detector for use in detecting a human in a 
o 0.4 ■ OA - 0.6 * * * 0.4* «iir(i <L32 W ) 20 scene, the detector comprising; 

a first camera sensitive to a first band of wavelengths 
wiihin a reflected infrared radiation range and operable 
to provide a first image output representative of the 
scene; 

a second camera sensitive to a second band of wave- 
lengths within the reflected infrared radiation range and 
operable to provide a second image output representa- 
tive of tbc scene; 
a fbsci connected to Sflid first and second carneras Opcr- 
- able un the first image output and the second image 
ontpul lo provide a fused image Output having an 
increased contrast between a human and a background 
in the scene; 

a Ihrcshnlder connected to said fuscr operable uu the 
fused image output for use in segmenting a detected 
human from the scene* the ibreshakler providing a 
thresbolded output; and 
an image indicator connected to said thresholder and 
operable on the 1b refolded output to detect a human in 
the scene. 

2. The detector of claim 1, wherein said first and second 
cameras have the same fieMs-of-vjcw. 

3. 'She detector of claim X wherein: 
said Ursi camera has its field of view focused on a first 

sensing array; and 
said second camera has its field of vjew focused OB a 
second sensing array. 

4. Tbe detector of cJaim 3, wherein: 
the first sensing array comprises mxn pixels; 
rhe second sensing array comprises mxn pixels; 
each pixel of said first sensing array is designs led as 

where i ranges from 1 to m and j ranges from 1 to o; and 
each pixel of said second sensing array is designated as 
(i Sh w bcrc i ranges from 1 lo m and j ranges from 1 
lo n. 

5. The detector of claim 4, wherein: 
a spectral power of a pixel from the first sensing array is 

designated as P(ij)j; and 
a spectral power of a pixel from tbe second sensing array 
is designated as j^- 

6. Tbe detector of claim 5, wherein: 
said fiiser fuses the spectral power P(ij), of each pixel of 

the first sensing array with the spectral power P(ij), of 
each corresponding pixel of I he second sensing array; 
and 



Consequently, the radiant power on the camera 11, 12 
pixel will be 



= J. 4^-1 0" S t O.O06 
= 0.084*10-" Wofe 

The camera's detectivity D* is D'-IO 12 cm vHx/Watis. 
The noise equivalent power (NEP) is relaled to detearviiy 
D», pixel area A, and electronic bandwidth Af by tbe 
following equation 



4* 



The bandwidth Af is determined by the exposure time of 
camera ll p 12- 'Jhe exposure lime depends un vehicle 41 
velocity 42, camera range 40, and the camera U, 12 field of 
view such that the images smear less than 1 pixel. Assuming 50 
vehide 41 traveling at a Speed of 65 mph, ai a distance d <G 
of 40 meters (m) away from camera H, 12, and with a field 
or view of 1.6 m, ihe 320x240 pixel array of camera 11, 12 
gives a maximum exposure time of 1 ms or a bandwidth of 
Af=l kHz_ Substituting tbe values for A, Af, and D* in the 55 
formula of NEP, one gels 

NEfl-lJsnrr u wate. 



60 



Therefore, the signal to noise ratio S/N will be 

S/N-0WN*P>=O.7. 

In conclusion, assuming a worst cuse scenario (overcast 
day, dirty windshield, dark occupant skin) one determines 
that camera U, 12, equipped wiih a SJ1 lens, a 1.4 to 1-7 /mi 65 
filter, and a polarizer, if it is positioned at a distance 43 of 
d«40 m from incoming car 41 and at a height 44 of h»7 m 
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an output of said fuscr is a spectral power of each fused 
pixel P(i jVwat is a fusion of the $i>eciral powers Ppj^ 

«*P(Hi)>- 

7. The detector of claim 6, wherein: 

C ia a constant- 

8. The detector of claim 7, wherein said ihrc&holdcr 
proCCSSCS the spectra) power P(ij)/Of each fused pixel into 
a spectral power ftf j), of a thrcsbold pixel. 

9. The detector of claim fc, wherein: 

F(ij) r is equal to a first value or a second value; 
the first value is a minimum spectral power; and 
the second value is a maximum spectral power. 

10. The detector of claim 9, wherein: 

Ff^ij), is equal to the first value if P^ is below a 

threshold value; and 
T^i^, is equal to ibe second value if P (id)/ is above the 

threshold value 
U. The detector of claim 10, wherein a wavelength 
between the first and second bands of wavelengths is about 
1.4 microns. 

12. The detector of claim 11, wherein: 

a histogram reveals a numerical distribution of pixels over 
a range of Spectral powers F (ij^for each frame; and 

Ihc threshold valve js a spectral power selected between 
two most populous distributions of pixels od the his- 
togram. 

13. The detector of claim 12. wherein C is selected 
according to lighting, conditions of a scene within the 
fields-of-view. 

14. The detector of claim 13, further comprising an image 
indicator connected lo said Ihresholdcr. 

J5. The detector of claim 14, wherein: 
the first band of wavelengths is between about 0.S and 1.4 
microns; and 

the second band of wavelengths is between about 1.4 
microns and 22 microns. 

16. A human detector comprising; 

first camera means fox sensing radiation of a scene within 
a first band of* wavelengths in a reflected infrared 
radiation range; 

second camera means for sensing radiation of the scene 
within a second band of wavelengths in the reflected 
infrared radiation range; 

fusing means connected to said first and second camera 
mean* for fusing spectral powers P(i j) T of pixels (ij) 3 
from said first camera means with spectral powers 
P(i,x) 2 of corresponding pixels from said second 
camera means, resulting in fused spectral powers P(i, 
j)^and 

thresholding means, connected u> Slid fusing means, for 
thresholding fused spectral powers P(ij)/ from said 
fusing means, resulting in threshold spectra 1 powers 
P(i j)- and 
wherein: 

the scene is focused by said First Camera means cm a first 
sensing array having mxn pixels; 

the SCCftC is focused by said second camera means on a 
second sensing array having mxn pixels; 

each pixel is located at ith rnw and jth cohimn; 

Origins; and 

0<j£u. 

17. The detector of claim .16*, further comprising indicat- 
ing means for displaying a segmented image of the scene. 



25 
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18. The detector of claim 16, wherein: 
P(ij) r P(i jX-C'FtfJfe and 

C is a constant del ermine d according 10 a lighting con- 
ditio]] of the scenc- 

19. The detector of claim 18, wherein: 
POJ^V.iffXij^T; 

P(io>v 2 irp(ijVr; 
V a is a firai value; 
v a is a second value, and 
T is a threshold value. 

20. The detector of claim 19, wherein T is deterrnined by 
a special power distribution of fused pixels (i j)^ 

21. Tbo detector of claim 20, wherein a wavelength 
between the first and second bands of wavelengths ia about 
1.4 microns. 

22. 1 "he detector of claim 21, wherein: 
Ibe first band Of wavelengths is between about 0.8 and 1 .4 
micron*; and 

the second band of wavelengths fs between about 1.4 
microns and 2.2 microns. 

23. A method for delecting humans, comprising: 
focusing a scone on a first mxn pixel array that is sensitive 

to light of a first bandwidtll Wilbin a reflected infrared 

radiation range; 
focusing the scene on a second mxn pixel array that is 

sensitive to light of a second bandwidth within the 

reflected infrared radiation range; 
fusing a spectral power of each pixel of the firsr 

mxn pixel array with a spectral power P(ij)u of each 

corresponding pixel of tbc second mxn pixel array, to 

result in a spectral power P(ij)/ of a fused pixel, 

respectively; 

thresholding the Spectral power PCij^of each fused pixel 

inio a first value if the spectral power is greater ibao a 

threshold value; and; 
thresholding the spectral power P(ij^> of each fused pixel 

into a second valve if the spectral power is less than the 

threshold value; 
wherein: 
Oci^m; and 
fXj^n. 

24. The method of claim 23, wherein: 
^^/-Pd^i-C^PCij^ and 

C is a constant dependent upon a lighting condition of the 
scene. 

25. The method of claim 24, wherein: 
P(yX-V a ifP(id>T; 
PCia-VaifPCij^T; 

V i is a first value; 

V 2 is a second vajue; and 

T is a threshold value dependent Upon a distribution of 
spectral powers flp dV °£ lbe ftwcd pixels. 

26. The method of claim 25, wherein a wavelength 
between the first and second bandwidth 5 is about 1.4 
microns. 

27. The method of claim 26> further comprising display- 
ing a segmented image with the first and second values of 
the thrcsbold pixels (ij) r 

28. The method of claim 27, wherein: 

the first bandwidth is between about 0.8 and 1.4 microns; 
and 

the second bandwidth is between about 1 .4 microns and 
2J1 microns- 
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29 AmcOiodf6rdeU«liDgliqi»ansinas«nc,Lhemelhod 33. The method of claim 29, wherein a wavelength 
comprising: between the first and second ba^dwidths is abouL 1.4 

focasiQEascenconafifstm^pi^l^caythaHHscnsiTivc microns. . 
to lighl of a first bandwidth within a reflected infrared 34. Tbc method of claim 29, wherein ihe Or&l bandwidth 
radiation range to provide a first image outpui rcpre- 5 is between about 0.8 and 1.4 microns and ihc second 
seniativc of the scene; bandwidth is between about 1.4 microns and 2.2 microns, 
focusing ihc scene on a second mxn pixel array thai is 35. The method of claim 29, wherein a difference: in 
SsnsJrive to li$bl of a second bandwidth wilhin the reflectance for human fikin m the first bandwidth relative lo 
reflected infrared radiation range to provide a second the second bandwidth is greater than a difference in reflcc- 
image output representative of the scene; and 1Q tancc lor objects other than human skin in ihc first band- 
ying Lbe first and second image outputs lo delect a human width reUtive to the second bandwidth, 
in the scene. 36. The method of claim 35, wherein using the first and 

30 The method or claim 29, wherein using the Unit and second image outputs to deiect a human in lb© scene 
second image outputs to detect a human in the scene s comprises performing a weighted difference calculation of 
comprises fusing spectral power of The pfrcb or the firsi " spectral power of ihe pixels of ihe first image output with 
image output with spectral power of corresponding pixels of Spectral poweT of corresponding pixels of ihe second image 
the second image output to provide a fused image numut output resulting in a weighted difference outpui for the 
having an increased contrast between a human and a back- pixels. 

ground in the scene. 20 37 - The method of claim 3<s wherein using inc first and 

31. The method of claim 30, wherein using the first and second image outputs to delect a human in ibe scene runner 
second image outputs to detect a human in the scene further comprises comparing the weighted difference outpui far the 
comprises segmenting the human from the scene by com- pixels lo one or more threshold values to differentiate pixels 
paring the Spectral poweT of each pixel of the fused image representative or human skin from pixels representative of 
outpui to one or more threshold values. ^ Objects other than human skin. 

32. The mcihDd of claim 31, wherein me method further 

comprises displaying a segmented image. * * * * * 
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